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Executive Summary 


EXECUTIVE SUMMARY 

As one of the parental choice provisions implemented with Title I funds under the No Child 
Left Behind Act of 2001 (NCLB), parents of low-income students in low-performing schools are 
offered a choice of Supplemental Educational Services for their children. Supplemental 
Educational Services (SES) include tutoring or other academic support services offered outside 
the regular school day, at no charge to students or their families, by public or private 
organizations that have been approved by the state as SES providers. School districts are 
required to offer SES to all students from low-income families attending a Title I school that for 
three consecutive years did not make Adequate Yearly Progress (AYP) toward meeting state 
standards regarding the percentage of students (or subgroups of students) who have achieved 
proficiency in reading and math. 

Implementation of SES has grown quickly and is now widespread. The number of students 
participating more than doubled (to 530,000) from 2004-2005 to 2006-2007 (Stullich et al. 
2009) and continued growing (to 672,101) in 2008-2009 (www.eddataexpress.ed.gov) . Whether 
ESEA will require SES in its next reauthorization is unclear. The Department of Education’s 
blueprint for reauthorization of ESEA, released in March 2010, recommends that chronically 
low-performing schools should no longer be required to fund SES but instead should be required 
to implement any of a number of “data-driven interventions,” which could include “expanded 
learning time, supplemental education services, public school choice, or other strategies” 
( http://www2.ed.gov/policv/elsec/leg/blueprint/blueprint.pdD . 

Over the years, SES funding in a few districts has fallen short of what was needed to serve 
all eligible youth who were interested in accessing services. Federal regulations governing the 
implementation of SES require prioritization of services to the lowest achieving eligible students 
when resources are constrained and funds are insufficient to provide SES to all eligible students 
who request services (Section 1 16[b] [10] [C] ; 34 CFR 200.45 [d]). A few prior studies have 
estimated the effect of SES for students receiving services and provide some preliminary 
evidence of the effectiveness of these tutoring services in raising student achievement (for a 
summary of some of these studies, and a discussion of implementation issues, see Springer et al. 
2009b). 

This report presents the findings of an evaluation sponsored by the Institute of Education 
Sciences (IES) at the U.S. Department of Education (ED) and conducted by Mathematica Policy 
Research (Mathematica) that uses a regression discontinuity (RD) design to assess the potential 
benefits of offering SES in districts that have unmet need. Specifically, the study focuses on six 
school districts in which more eligible students applied for SES than could be served with 
available funds (i.e., oversubscribed districts), and which therefore allocated scarce SES spaces 
by giving priority to lower- achieving students among the eligible applicants. The current study is 
the first to use a research design that can directly account for selection into services and therefore 
(if reasonable assumptions are met) support drawing causal inferences about the impact of SES 
on students at the cutoff of receiving services (Hahn, Todd, & Van Der Klaauw, 2001; Shadish, 
Cook, & Campbell, 2002; Imbens & Lemieux, 2008). The study’s RD design estimates the 
impact of SES on academic achievement by comparing the post-test scores of students offered 
services with those for students whose prior achievement scores were slightly too high to be 
offered services, with the comparison adjusting for the prior achievement scores that were used 
to determine assignment to services. The estimates apply to students in the study’s school 
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districts who are on the cusp of receiving services. Because different cut points were used in 
different grades across the six districts, these estimates show the effect on students on the cusp of 
participating in the program in multiple contexts in oversubscribed districts. 

Research Questions 

The evaluation study’s design and data collection plans were developed to provide the most 
methodologically rigorous answers to the study’s primary research question: 

1. What is the average impact of offering SES to eligible applicants who are on the cusp 
of having access to services, in school districts where services are oversubscribed? 

The study also addresses an additional research question: 

2. What are the characteristics of SES provided to students in oversubscribed districts? 

Are the characteristics of services, providers, or practices in host school districts 
correlated with the estimated impacts? 

Key Findings 

Findings from this evaluation are based on six, nonrepresentative school districts in three 
states (Connecticut, Ohio, and Florida) where more eligible students applied for SES than could 
be served with available funds. Across the six oversubscribed districts included in our study, 
there were 50,843 applicants to SES, of which 30,673 were in study grades (3-8). Among these 
applicants, 24,113 were assigned to services based on a measure of prior achievement. These 
24,113 constitute the study population. Among the students in the study population, 
19,750 students (82 percent of the applicants) were offered SES based on their prior scores while 
4,363 students were not offered SES, and 16,954 (86 percent of those offered services) 
participated in SES (i.e., reported any tutoring services). Almost all participating students 
(98 percent) were offered services with their first choice providers, with most students being 
served by a few dominant providers. 

The answers to the key questions addressed in the study are as follows: 

• What is the average impact of offering SES to eligible applicants who are on the 
cusp of having access to services, in school districts where services are 
oversubscribed? Across the six oversubscribed districts, we find no evidence of 
impacts of offering SES to students near the cut point for an offer. For students in 
these oversubscribed districts in grades 3-8 at the cusp of receiving an offer of 
services, we find no statistically significant impact of offering SES on student 
achievement in reading or in mathematics. 1 The point estimate of the average impact 
on reading is -0.03 standard deviations, and that for mathematics is 0.05 standard 
deviations. Furthermore, there is no evidence of potential benefits for at-risk 


1 Note, however, that the study did not explicitly test the expansion or contraction of SES. Estimated impacts 
are based on the assumption that the number of students offered services would change at the cutoff while all other 
relevant factors are unchanged. 
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subgroups of students. Similarly, we find no statistically significant impact of 
participating in SES on student achievement in reading or mathematics. The 
estimated impact of participating (which involved an average of 21 hours of services) 
is -0.10 standard deviations for reading and 0.11 standard deviations for math (again 
estimated for students in grades 3-8 near the cutoff for an offer, in these 
oversubscribed districts). 

• What are the characteristics of SES provided to students in oversubscribed 
districts? Across districts participating in this study, services averaged 21.2 hours per 
student for the school year (standard deviation of 8.8), with over a third of the 
students (36 percent) receiving tutoring in both reading and math, 55 percent 
receiving tutoring in only reading, and 9 percent receiving tutoring in only math. For 
students receiving reading services, the mean was 17.2 hours of tutoring (standard 
deviation of 9.2). For students receiving math services, the mean was 12.5 hours of 
tutoring (standard deviation of 8.2). In the study districts, most providers (70 percent) 
were for-profit firms. On average, 60 percent of providers’ instructional staff were 
regular schoolteachers working in the local district. Most providers (64 percent) 
offered services at the schools of their students. Providers reported that group sizes of 
2-5 students were most frequently used, with most other sessions provided 
individually (in one-on-one sessions). On average, 44 percent of provider services 
were in groups of 2-5 students, 34 percent in one-on-one sessions, and 21 percent in 
groups of 6-10. 

• Are the characteristics of Supplemental Educational Services, providers, or 
practices in host school districts correlated with the estimated impacts? There is 
variation across providers in the average number of hours of math and reading 
services received by students, with average hours in services focused on math ranging 
from 0 to 27 across providers and average hours in services focused on reading 
ranging from 0 to 43. However, the intensity of services is not significantly related to 
the estimated size of impacts on math or reading achievement (for these eligible 
students near the cutoff for having been offered SES). We also found no evidence that 
any other observed provider characteristics were significantly associated with 
stronger impacts. 

Background on Title I SES 

Districts are required to offer SES to all low-income students in schools that have fallen 
short of Adequate Yearly Progress (AYP) proficiency standards for a third time. (After missing 
AYP for two consecutive years, schools receive the “identification for improvement” 
designation.) As one of the options available to parents under NCLB, the SES program permits 
eligible families to choose from a wide variety of state-approved SES providers, including 
national for-profit firms, local nonprofits, faith-based organizations, institutions of higher 
education, and local school districts. School districts are permitted to become approved providers 
unless they are themselves identified for improvement under NCLB. As of September 2010, 
more than 3,000 providers across the country had been approved to offer SES. 

Federal regulations govern the funding of SES. State education agencies allocate Title I 
funds to districts with eligible students. Those districts are then required to make available up to 
20 percent of their total Title I, Part A funds for a combination of SES and transportation for 
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students using NCLB’s school-choice option. In addition to regulations for district- wide SES 
expenditures, there is a restriction on the district’s maximum expenditures per student. NCLB 
requires provision of SES to all eligible students if expenditures are less than 20 percent of their 
Title I, Part A funds. However, if funds are insufficient to provide SES to all students who 
request them, non-regulatory guidance from the U.S. Department of Education (ED) indicates 
that priority must be given to the lowest-achieving students ( Section 1116[b][10][C]; 34 C.F.R. 
§200.45 [cl] ) based on “fair and equitable procedures in determining which students are the 
lowest achieving.” This prioritization rule creates the opportunity to conduct a study using an RD 
design in oversubscribed school districts. 

What Kinds of Districts and Students Were Included in the Study? 

Oversubscription for SES is unusual among school districts. In 2008-09, when we were 
drawing our study sample, ED’s Office of Innovation and Improvement (Oil) identified only 
forty school districts that might have more eligible applicants for SES than they could serve. We 
ultimately found only nine of these assigned districts that were actually oversubscribed and that 
intended to ration scarce spaces based on a “cut-off’ value on students’ prior year’s test score. 
Eight of the nine districts agreed to participate, of which two districts were subsequently 
excluded from the study because assignment practices did not support the RD design. 

The study’s six school districts are by no means nationally representative. They are in three 
states: Connecticut, Ohio, and Florida. As a result of statewide rules regarding unspent SES 
funds, Florida districts had a stronger incentive than districts in other states to ensure that as 
many students as possible were participating in SES. Unlike most other states at the time, Florida 
had in place funding rules that did not permit unused SES funds to be automatically returned to a 
school district’s overall Title I allocation ( http : //www. fldoe . or g/flbp so/pdf/finalpro .pdf) . Aside 
from being oversubscribed for SES, the six study districts also differed from the average 
characteristics of all Title I school districts in having higher proportions of economically 
disadvantaged students. Study districts were largely comprised of urban and suburban schools 
(38 percent and 56 percent, respectively), with 66 percent of district students eligible for 
free/reduced-price lunch (as compared with 56 percent of students in all Title I schools 
nationally). These six districts, in comparison with Title I schools nationwide, also had higher 
percentages of black students (34 percent versus 20 percent) and Hispanic students (46 percent 
versus 28 percent). 

Across the six study districts, applicants to SES numbered 50,843 students, 30,673 of whom 
were in study grades (3-8). Of those, 24,113 were assigned to treatment and comparison groups 
using a continuous measure of prior achievement (while the rest were in different categories, 
such as prioritized grade levels in which all eligible applicants were admitted). These 
24,113 students constitute the study population. Among the study population, 19,750 students 
(82 percent) were offered SES based on their prior scores, 4,363 students were not offered SES, 
and 16,954 (86 percent of those offered services) participated in SES (i.e., reported any tutoring 
services) Parents of eligible students were given a choice of approved providers, and the 
overwhelming majority of students (98 percent) were assigned to their first choice. 


2 See http://www2.ed.gov/nclb/choice/help/ses/guidance.html for non-regulatory guidance. 
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Five percent of SES participants in our sample were white, 47 percent were black, and 
46 percent were Hispanic. Sixteen percent of participants were English language learner (ELL) 
students, and 21 percent of participants were students with a disability (those in special 
education). 

What Were the Study’s Data Sources? 

The data collection had two distinct sources that are integrated to answer the study’s 
research questions. The first is student-level data on all SES applicants from the districts, 
including demographics, SES provider choices, number of hours of tutoring received, and 
standardized test scores in math and reading from 2007, 2008, and 2009. Though collected from 
districts, the standardized tests used as outcomes are the same tests used for state accountability 
purposes. The second component is a survey of SES providers in the six districts that gathered 
information on provider characteristics and practices. 

What Is the Potential Achievement Benefit of Offering SES to Applicants at the 
Cusp of Eligibility in Oversubscribed Districts? 

This study uses an RD analysis to assess the effectiveness of SES implemented in real-world 
settings among school districts that lacked the funds needed to serve all SES applicants eligible 
for services. 

Applications for SES were received by districts in the fall of 2008 and students received 
services during the 2008-2009 school year. Students were offered SES based on a continuous 
measure of academic need (such as a state achievement test score). In each district, students with 
prior test scores below a specific cutoff value were offered SES (the treatment group), whereas 
students above the cutoff value were not offered SES (the comparison group). Under this design, 
estimates of the impacts of SES on students at the cutoff for being offered services can be 
obtained by comparing the outcomes of students just below and above the cutoff value, after 
adjusting for the score on the test used to assign students to treatment and comparison groups 
(see, for example, U.S. Department of Education 2010; Shadish, Cook, & Campbell, 2002). 
Cutoff scores varied across districts and often across grade levels within districts, creating 
42 “mini- studies” with unique cut points. Because different cut points were used, these estimates 
show the effect on students in multiple contexts (assuming other factors are unchanged). 

The study had an estimated 80 percent or greater chance of detecting an overall impact of 
SES if the true impact is 0.12 standard deviations or larger. An impact of 0.12 standard 
deviations is equivalent to moving from the 50th percentile to the 55th percentile on a test with 
normally distributed scores. 

For our primary impact analysis, we found no statistically significant effect of offering SES 
to eligible applicants near the cutoff, either on reading scores or math scores. Point estimates 
were -0.03 standard deviations in reading and 0.05 standard deviations in math, but neither 
approached statistical significance. 

Because not all students with prior scores below the relevant cutoff value actually received 
services (2,796 students, or 14 percent of those students offered SES, did not participate), we 
also estimated the impact of participation. For applicants near the cutoff, there were no 
statistically significant impacts of participation in SES on either subject. Other variants of the 
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impact of participation were estimated using alternative definitions of SES participation, 
including the impact on the specific subject in which services were received. We found no 
statistically significant impacts. 

What Were the Key Characteristics of Services, and Were Any Characteristics 
Related to Estimated Impacts? 

The study examined characteristics of SES providers and the services they provided to 
students. Understanding the variation in provider services and students’ access to SES is 
important context for understanding the impact analyses. In addition, we capitalize on variation 
in program implementation across districts and providers to look at the relationship between such 
characteristics and program impacts. None of these relationships were stronger than what we 
would expect to see by random chance alone. 

The first descriptive question of interest relates to the intensity of services. In fact, over the 
course of an academic year, providers reported that a complete course of services was about 
28 hours, on average. This is substantially less than the 45-hour average found in a previous 
survey of providers in 16 districts across the country (Vernez et al. 2009). The length of a full 
course of services varied across providers, from 13 hours to 108 hours, with approximately 
70 percent of providers reporting that a full course of services was 30 hours or less. However, 
there was no statistically significant relationship found between the length of the course of 
services and the estimated impacts of provider services on students at the cutoff for receiving 
SES. 

Average student participation in our sample was even less — at 21.2 hours — than the average 
full course. Across the six districts, 36% of SES participants (6,058) received tutoring in both 
math and reading, 9 percent (1,560) received tutoring in only math, and 55 percent (9,336) 
received tutoring in only reading. For students who received tutoring in math, mean math hours 
were 12.5 (standard deviation of 8.2). For students who received tutoring in reading, mean 
reading hours were 17.2 (standard deviation of 9.2). Thus, time spent in SES was only a small 
fraction of the time that students typically spend in school. For example, if a student received 
12.5 hours of SES math instruction (the mean hours received), it would equal approximately 
7 percent of a full year of math instruction, assuming that students spend one hour a day on math 
as part of their regular curriculum. Likewise, if a student received 17.2 hours of reading 
instruction, it would equal approximately 10 percent of a full year of reading instruction. The 
average number of hours of services that students received was not associated with the estimated 
impacts of provider services on students at the cutoff for receiving SES. 

Various other characteristics of services and providers were potentially relevant to their 
effectiveness in raising student achievement, and therefore potentially of interest to state 
authorizing agencies and to the Department of Education. For example, provider size varied 
widely. There were a few dominant providers in each district; in our sample, 9 providers served 
50 percent of the students, with the remaining half of students served by the remaining 
179 providers. Providers had, on average, at least two years of experience providing SES or other 
educational services. The majority of providers (69 percent) were for-profit organizations that 
relied extensively on local school teachers to serve as their SES instructors. Almost all services 
were provided face-to-face (92 percent of providers), most commonly in small group sessions of 
2 to 5 students (44 percent in groups of 2-5; 34 percent in one-on-one sessions; and 21 percent in 
groups of 6-10). Larger providers were more likely to rely on slightly larger group sessions (6 to 
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10 students). A majority of SES providers (63 percent) administered multiple diagnostic 
assessments to students. Although providers reported their programs to be aligned with state 
standards, they typically were unable to identify the particular instructional programs used in the 
schools of the local district. 

We examined relationships between provider characteristics and impacts, but none of these 
relationships were stronger than what we would expect to see by random chance. 
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